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Abstract 

Numerous definitions for complexity have been proposed over the last 
half century, with little consensus achieved on how to use the term. A 
definition of complexity is supplied here that is closely related to the Kol- 
mogorov Complexity and Shannon Entropy measures widely used as com- 
plexity measures, yet addresses a number of concerns raised against these 
measures. However, the price of doing this is to introduce context de- 
pendence into the definition of complexity. It is argued that such context 
dependence is an inherent property of complexity, and related concepts 
such as entropy and emergence. Scientists are uncomfortable with such 
context dependence, which smacks of subjectivity, and this is perhaps the 
reason why little agreement has been found on the meaning of these terms. 



1 The problem of Complexity 

In the last 15 years, the study of Complex Systems has emerged as a recognised 
field in its own right, although a good definition of what a complex system 
actually is has eluded formulation. Attempts to formalise the concept of com- 
plexity go back even further, to Shannon's inception of Information Theory ^^ . 
A good survey of the tortuous path the study of complexity has followed is 
provided in Edmonds Q. Of particular importance to this paper is the concept 
of Kolmogorov Complexity (also known as Algorithmic Information Complexity) 
introduced independently by Kolmogorov|lC|, ChaitinQ and Solomonoff fl5f . 
Given a particular universal Turing Machine]^ (UTM) U, the Kolmogorov com- 
plexity of a string of characters (ie a description) is the length of the shortest 
program running on U that generates the description. 

There are two main problems with Kolmogorov complexity: 



1 A Turing Machine is a formal model of a digital computer 
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1. The dependence on U, as there is no unique way of specifying this. Even 
though the Invariance theoremp], Thm 2.1.1] guarantees that any two 
UTMs U and V will agree on the complexity of a string x up to a constant 
independent of x, for any descriptions x and y, there will be two machines 
U and V disagreeing on whether x is more complex that y, or vice-versa. 

2. Random sequences have maximum complexity, as by definition a random 
sequence can have no generating algorithm shorter than simply listing the 
sequence. As Gell-Mann[|| points out, this contradicts the notion that 
random sequences should contain no information. 

The first problem of what reference machine to choose is a symptom of con- 
text dependence of complexity. Given a description x, any value of complexity 
can be chosen for it by choosing an appropriate reference machine. It would 
seem that complexity is in the eye of the beholder |J. Is complexity completely 
subjective? Is everything lost? 

Rather than trying to hide this context dependence, I would prefer to make 
it a feature. Instead of asserting complexity is a property of some system, it is 
a property of descriptions (which may or may not be about a system). There 
must also be an interpreter of these descriptions that can answer the question of 
whether two descriptions are equivalent or not. Consider Shakespeare's Romeo 
and Juliet. In Act II, ii, line 58, Juliet says "My ears have yet not drunk a hun- 
dred words" . If we change the word "drunk" to "heard" , your average theatre 
goer will not spot the difference. Perhaps the only one to notice would be a 
professional actor who has played the scene many times. Therefore the different 
texts differing by the single word "drunk/heard" in this scene are considered 
equivalent by our hypothetical theatre goer. There will be a whole equivalence 
class of texts that would be considered to be Romeo and Juliet by our theatre 
goer. 

Once the set of all possible descriptions are given (strings of letters on a page, 
base pairs in a genome or bits on a computer harddisk for example), and an 
equivalence class between descriptions given, then one can apply the Shannon 
entropy formula to determine the complexity of that description, under that 
interpretation: 

C(x)= limflogatf-logaW&aO (1) 

i— >oo 

where C(x) is the complexity (measured in bits), £(x) the length of the de- 
scription, TV the size of the alphabet used to encode the description and cu(l, x) 
the size of the class of all descriptions of length less than I equivalent to x. 
We assume that the interpreter of a description is able to determine where a 
description finishes, so that a description y of length i(y) is equivalent to all 
j^t—Ky) length ( descriptions having y as a prefix. 

If we choose our description set to be all bitstrings, and our equivalence class 
to be all bitstrings that produce the same output when executed by a universal 
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Turing machine U, then 

u(t,x)= 2 ^ (P) > ( 2 ) 

p:U(p)=U(x),e(p)<£ 

where £(p) is the length of program p. As £ — > oo, this distribution (when 
normalised) is known as the universal a priori probability^^, Def 4.3.3]. By 
the coding theorem, the complexity defined by equation (Q) is equal to the Kol- 
mogorov complexity up to a constant independent of x. 

This perspective helps us understand the problem of random strings having 
maximal complexity. In an equivalence class generated by a human observer, 
one random string is pretty much the same as any other. Therefore the w term 
of a completely random string will large, probably of comparable size to N . 
Therefore the complexity of a random string, as interpreted by a human observer 
is low, exactly the property required of Gell-Mann's effective complexity. 

Whilst context dependence would appear to open up the curse of subjectivity, 
it needn't necessarily do so. In many situations, the equivalence relation is well 
defined. For example the notion of species in biology is reasonably well defined 
(although disagreement exists in a number of cases). This could, in principle, 
along with a detailed knowledge of the genetic code, be used to estimate the 
complexity of different species. This principle has been used in a number of 
artificial life studies [ll| for studying the evolution of complexity. 



2 Emergence 

Emergence is that other area of complex systems study that has experienced 
controversy and confusion. Its importance stems from the belief that emergence 
is the key ingredient that makes complex systems complex. Putting things 
colloquially, emergence is the concept of some new phenomenon arising in a 
system that wasn't in the system's specification to start with. There is some 
considerable debate as to how this happens, or whether emergence can truly 
happen within a formal system such as an agent-based model [fl3|, 

Let me illustrate this debate with the example of gliders in The Game of Life. 
I would contend that this phenomenon is emergent, in the sense that the glider 
concept is not contained within the cellular automaton implementation language 
- namely states and neigbour hoods. This puts me at odds with Rosen, who 
would argue that gliders are but complicated combinations of simple machines 
(the cellular automata), not examples of complexity 

What is my definition of emergence then? To set the scene, let me introduce 
two descriptions of a system, called the microdescription and macrodescription, 
each coded in their own language. Ronald et al., use the term C\ and Li to refer 

2 Rosen uses the word complexity as a quality, rather like emergence is used in this paper, 
as opposed to a quantity. 
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to the micro- and macrolanguage respectively [[l2|. They call the microlanguage 
the "language of design" in view of artificial life applications. However, the 
microdescription may equally well be our best description of what happens at 
the most fundamental level. To make it clear, emergence is not due to the 
failure of the microdescription as a modeling effort, since the emergent property 
should still appear as the result of a computer simulation constructed using the 
microdescription. 

We also assume that the macrodescription is a good theory. There is, in 
general, a trade-off between the predictive power of a theory, and its explanatory 
power. Of course, a theory may be neither predictive nor explanatory, but in 
this case the theory is not so good, and would be rejected in favour of one that 
is better. One can, of course, produce examples of emergent concepts based on 
a poor macrodescription, but in this case the correspondence with the real or 
simulated system would be lacking, and the concept would not be an observed 
phenomenon. 

An emergent phenomenon is simply one that is described by atomic concepts 
available in the macrolanguage, but cannot be so described in the microlanguage. 
In the case of the glider in The Game of Life, any attempt at describing a glider 
would involve the CA transition table (naturally), but also the specific pattern 
of cell states that make up the glider. But which pattern? A glider can appear 
at any location within the CA, and may have one of four possible orientations. 
The description cannot represent the fact that two gliders separated diagonally 
by 1 cell in along each axis with the same orientation are temporally related. A 
glider, as an object-in-itself, is a pure macrodescription object. 

Ronald et al. focus on the element of surprise as a test of emergence. In 
this, they are trying to capture emergence as some kind of dissonance between 
the micro and macro languages. To be fair to their work, they claim only a 
test for emergence, not a definition, along the lines of the more famous Turing 
test. However, the surprise factor really only works when the macrolanguage 
is enlarged (by the emergent concept) in order to make the macrodescription a 
better model of the system. Once the emergent property has been recognised by 
the observer, the property is no longer surprising. The definition of emergence 
given here works, regardless of whether the observer is still surprised or not. 

Of considerable interest is, given a system specified in its microlanguage, 
does it have emergent properties? There is no general procedure for answering 
this question. One has to construct a macrodescription of the system. If this 
macrodescription contains atomic concepts that are not simple compounds of 
microconcepts, then one has emergent properties. Is there a best macrodescrip- 
tion for any given system? This question is outside of the scope of this paper, 
and needs to be answered by theories of how scientific theories are developed. 
However, in general, it seems unlikely that there would be a "best description" , 
as it depends on the motives of the person using the description. For example, 
I have already alluded to the tension between predictive power and explanatory 
power. 
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3 Entropy as a case study of emergence 



Thermodynamics provides an excellent case study in emergence, as it is well 
understood theory, and also illustrates the link between emergence and com- 
plexity. 

Thermodynamics is a macroscopic description of material systems, expressed 
in terms of concepts such as temperature, pressure and entropy. It is related to 
the microscopic description of molecular dynamics via the reductionist theory 
of statistical mechanics. 

Within thermodynamics, entropy is defined by differences along a reversible 
path: 

AS = AQ/T 

Within the framework of thermodynamics, this quantity is objectively defined, 
up to an additive constant (usually assumed to be such that entropy vanishes 
at absolute zero). 

Within Statistical Mechanics, entropy in the microcanonical ensemble is 
given by the Boltzmann formula: 

S = k B lnW, (3) 

where Ub is the Boltzmann constant (giving entropy in units of Joules per 
Kelvin), and W is the number of microstates accessible to the system for a 
particular macrostate. This formula is very similar to the information-based 
complexity formula (Q), and this led Jaynes|| to remark that entropy "measures 
our degree of ignorance as to the unknown microstate" . Denbigh and Denbigh || 
are at pains to point out that entropy is fully objective, as it only depends on the 
macroscopic quantities chosen to define the macroscopic state (the temperature, 
pressure and so on), and not dependent on any extraneous observer related 
property. Provided two observers choose the same microscopic and macroscopic 
description of the system, they will agree on the value of entropy. By my 
previous definition of emergence, entropy is an emergent quality of the system. 
This example illustrates how context dependent emergent properties can be fully 
objective. 

With entropy defined by (||), the well known H-theorem holds. As a con- 
sequence, the macroscopic (thermodynamic) description is time irreversible, 
whereas the microscopic description is reversible. Time irreversibility is like- 
wise an emergent property of this system. 

The very clear relation between the Boltzmann-Gibbs entropy (||), and com- 
plexity (Q) indicates that complexity is itself an emergent concept. If the mi- 
croscopic language and macroscopic language were identical, corresponding to a 
situation of no emergence, complexity of descriptions degenerates to the trivial 
measure of description length. 
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